Due to the rapid progress in the technologies for obtaining and storing digital images for diagnostic purposes in medicine (from photography over digital radiography to functional MRI

In today’s medicine, several diagnostic tasks are particularly difficult and are plagued by high interand intraobserver variability. For instance, reading mammographs in modern Western medicine or tongue images (photographs or videos of the stuck-out tongue) in traditional Chinese medicine require particular skills and long experience. For these tasks, image-based access to an archive with cases of known pathology would be a beneficial aid for both diagnosis and medical education. Thus, retrieving information stored in electronic healthcare records (EHR) based also on image patterns rather than only on alphanumerical indexing is required. Even modern EHR archives do not yet offer such augmented functionality. In this paper, we present a novel concept for an augmented medical information management (AMIM). Key functionalities of the concept are content-based medical image retrieval based on similarity of both the global image and local regions, capturing human expert knowledge with visual similarity metrics, retrieval from EHR archives and the Internet based on both visual and textual patterns, and a cheap and effective paradigm for home-based telemedicine. The latter functionality is based on a smart self-calibrating hand-held imaging device for standardized self-acquisition of tongue images. This idea is directly extendable to other diagnosis scenarios for wide-spread diseases, for example, screening for skin tumors. It will help significantly reduce the costs of healthcare systems and in particular improve the quality of life of (senior) citizens.